Skip to content

feat: OTel integration in XRPLD#7770

Draft
pratikmankawde wants to merge 1502 commits into
developfrom
pratik/otel-phase10-workload-validation
Draft

feat: OTel integration in XRPLD#7770
pratikmankawde wants to merge 1502 commits into
developfrom
pratik/otel-phase10-workload-validation

Conversation

@pratikmankawde

@pratikmankawde pratikmankawde commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

PR Chain: Phase-1aPhase-1bPhase-1cPhase-2Phase-3Phase-4Phase-5Phase-6Phase-7Phase-8Phase-9Phase-10
Base: develop

High Level Overview of Change

Adds full OpenTelemetry (OTel) integration to xrpld: distributed traces, native OTel metrics, and log/trace correlation, with a companion observability stack (Collector, Tempo, Prometheus, Loki, Grafana) and a workload harness that validates the end-to-end signal path.

Grafana Dashboards

Scope:

  • Tracing — spans on RPC, transaction, ledger, consensus, peer, and pathfinding paths, with W3C traceparent propagation across RPC and peer boundaries.
  • Metrics — native OTel meter API alongside the existing StatsD/beast::insight surface; spanmetrics derived by the Collector; Phase 9 gap-fill for previously unlabeled counters/timers.
  • Logs — trace_id/span_id injection into xrpld logs so Loki entries deep-link into Tempo.
  • Ops surface — provisioned Grafana dashboards (RPC, TX, ledger, consensus, peer, node health), alert rules with Slack/email routing, and docker-compose profiles for a 5-node workload.
  • Validation — synthetic RPC/TX/peer load generators plus a Prometheus/Tempo scripted checker that asserts every required span, attribute, and metric emits under load.

Context of Change

xrpld had only StatsD counters and free-form logs — no distributed traces, no structured attributes, and no way to correlate a slow RPC to the consensus round it landed in. This change introduces an OTel-native pipeline so operators can trace requests end-to-end, alert on structured SLIs, and jump from a log line to its trace.

Delivered in ten internal phases (plan docs under OpenTelemetryPlan/), landed here as a single feature branch off develop. Telemetry is off by default; enable via [telemetry] / [insight] server=otel in the config.

API Impact

  • Public API: New feature (new methods and/or new fields)
  • Public API: Breaking change
  • libxrpl change — adds telemetry primitives (SpanGuard, SpanNames, metrics registry) usable by libxrpl consumers; existing symbols unchanged.
  • Peer protocol change — none; traceparent rides existing peer message headers and is optional.

No wire-format or JSON-RPC schema changes. Config gains an opt-in [telemetry] stanza and an [insight] server=otel value.

Test Plan

  • Unit / GTest suites for telemetry primitives, propagator, and metric registry.
  • docker/telemetry/ workload compose brings up 5 validators + full OTel stack; make telemetry-workload-up runs the RPC/TX load generators.
  • Validation scripts poll Prometheus and Tempo to assert every required span/attribute/metric emits under sustained load; Grafana dashboards render non-empty panels.
  • Alert rules boot without env vars; Slack/email routing verified end-to-end.

pratikmankawde and others added 16 commits July 8, 2026 18:44
…XRPLF/rippled into pratik/otel-phase10-workload-validation
…cleaner)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(misc-include-cleaner)

std::ranges::transform/all_of come from <algorithm>, which is already included.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nclude-cleaner)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-tx-tracing

# Conflicts:
#	include/xrpl/telemetry/SpanGuard.h
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
@pratikmankawde pratikmankawde changed the title Pratik/otel phase10 workload validation feat: OTel integration in XRPLD Jul 9, 2026
@pratikmankawde pratikmankawde added DraftRunCI Normally CI does not run on draft PRs. This opts in. DistributedTracingAndObservability Distributed Tracing And Observability related changes labels Jul 9, 2026
@pratikmankawde

Copy link
Copy Markdown
Contributor Author

/ai-review

pratikmankawde and others added 5 commits July 9, 2026 15:01
- OTelCollector::formatName made static (no member state); braces added;
  <cctype> added for std::tolower.
- Telemetry.cpp getMeter(): braces around single-statement if.
- GetMeter.cpp: drop unused <opentelemetry/metrics/meter.h> and
  sync_instruments.h; add <xrpl/beast/utility/Journal.h> and <memory>;
  make sdkProvider const.
- ValidationTracker.h: add <cstddef> for std::size_t.
- detail/ValidationTracker.cpp: use std::views::reverse range-based loop;
  add <ranges>.
- test ValidationTracker.cpp: use auto for cast result.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues.

Review by Claude Sonnet 4.6 · Prompt: V15

pratikmankawde and others added 4 commits July 9, 2026 17:03
…ervice' to 'Service Name'

The template variable's Prometheus label name is `service_name` but the
UI display label read 'Service', which mismatched what users typed into
Grafana's Metrics Drilldown UI (which defaults to filter on a bare
`service` label). Rename the display label to 'Service Name' so it
unambiguously points at the underlying `service_name` Prometheus label.

Behaviour unchanged; template-variable name (`service_name`) and PromQL
label are the same. Only the widget label shown at the top of the
dashboard changes from 'Service' to 'Service Name'.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ervice' to 'Service Name'

Continuation of the same rename applied on phase-6-statsd (commit
91c6582): 5 more dashboards introduced on phase-7 (formerly the
system-* set, renamed to bare names by 2c590a4) still carried the
older 'Service' UI label. Aligning them with the rest.

Same reasoning as the phase-6 commit: the underlying Prometheus label
is `service_name`, and the UI label 'Service' misled users into typing
`service=xrpld` in Grafana Metrics Drilldown.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
pratikmankawde and others added 30 commits July 20, 2026 17:17
…hase1c-rpc-integration

# Conflicts:
#	include/xrpl/telemetry/Telemetry.h
…hase2-rpc-tracing

# Conflicts:
#	.github/scripts/levelization/results/loops.txt
…phase5-docs-deployment

# Conflicts:
#	include/xrpl/telemetry/Telemetry.h
…ve-metrics

# Conflicts:
#	include/xrpl/telemetry/Telemetry.h
#	src/libxrpl/telemetry/TelemetryConfig.cpp
Convert leading /// blocks in these 9 phase3-originating files to house
style /** */ (using the updated fix_doxy.py that now handles /// ->
block conversion). Comment-only: code is byte-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… refresh levelization

Convert leading /// blocks to house-style /** */ across the telemetry
files carried on this branch (using the updated fix_doxy.py). Also
regenerate levelization results. Comment-only: code is byte-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert leading /// blocks in LedgerSpanNames.h and PeerSpanNames.h to
house-style /** */. Comment-only: code is byte-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…consensus-tracing

# Conflicts:
#	include/xrpl/telemetry/TraceContextPropagator.h
#	src/xrpld/telemetry/PropagationHelpers.h
…phase5-docs-deployment

# Conflicts:
#	src/xrpld/telemetry/PropagationHelpers.h
xrpl_branch and xrpl_node_role are collector/infra-injected resource
labels used by perf-iac dashboards to identify the build under test and
its role in the perf cluster. They have no L1 span-attribute source or
L6 native-metric-label source (like the existing job/instance/job_type
builtins), so Rule D flagged them across every perf-iac dashboard.
Register them in the builtins set, matching the existing pattern for
infra-level dashboard labels. Add test coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g Rule D

xrpl_branch and xrpl_node_role are stamped by the perf-iac harness --
infrastructure outside this repo's OTel code -- so they have no L1
(*SpanNames.h), L2 (collector config), or L6 (MetricsRegistry.cpp) source
Rule D can derive them from. Unlike the generic builtins set (Prometheus/
Grafana mechanics every OTel setup has), these are repo-specific and
narrow, so they get their own EXTERNAL_INFRA_LABELS constant: a visible,
documented, deliberately narrow exception to the 'no hardcoded allowlist'
design principle, not a silent workaround. Add test coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ase9-metric-gap-fill

# Conflicts:
#	src/xrpld/telemetry/ValidationTracker.h
…ase10-workload-validation

# Conflicts:
#	src/xrpld/telemetry/MetricsRegistry.h
#	src/xrpld/telemetry/ValidationTracker.h
…h comments

An earlier Doxygen-style normalization pass on this file introduced a
stray xrpld_ prefix into 5 comment references to the validation
counters/gauge. The actual instrument names registered in
MetricsRegistry.cpp have no such prefix (validation_agreements_total,
validation_missed_total, validation_agreement). Comment-only fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DistributedTracingAndObservability Distributed Tracing And Observability related changes DraftRunCI Normally CI does not run on draft PRs. This opts in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant